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INTRODUCTION 


In recent years, there has been a rapid growth in the ability to obtain detailed data on large 
complex structures in three dimensions. This development occurred first in the medical field, with 
CAT scans and now magnetic resonance imaging, and in seismological exploration. With the 
advances in supercomputing and computational fluid dynamics, and in experimental techniques in 
fluid dynamics, there is now the ability to produce similar large data fields representing 3D struc- 
tures and phenomena in these disciplines. 

These developments have produced a situation in which currently we have access to data which 
is too complex to be understood using the tools available for data reduction and presentation. 
Researchers in these areas are becoming limited by their ability to visualize and comprehend the 3D 
systems they are measuring and simulating. 


HISTORY 


In response to this, there is growing activity in the area of visualization of 3D data. Some early 
work in this area was done by Harris et al. (1979) at the Mayo Clinic and Herman et al. (1984) at 
the University of Pennsylvania in the area of medical imaging. In 1983, Jaffey, Dutta, and 
Hesselink (1984) approached the subject from a different direction. They developed the "source- 
attenuation" model, and used holograms to visualize 3D subjects. More recently, there is stronger 
emphasis on interactive visualization, and concentration on techniques and systems for general use 
and commercial products (Goldwasser, 1985; Hunter, 1984). 

Much of the recent activity is directed toward improving and extending the use of graphics 
techniques for interactive visualization of data based on surface representations. The groundwork 
for this was done by Herman et al. Work in this area is continuing both in academic groups 
(Herman at the University of Pennsylvania (Herman et al., 1984 and Fuchs at North Carolina 
(Fuchs et al., 1985), and in several commercial ventures (notably CEMAX)). Also, graphics pro- 
jects at NASA, JPL, and aerospace corporations have been providing increasing support for visu- 
alization tasks based on conventional graphics concepts. 

The more interesting projects involve departures from conventional graphics. By careful use of 
transparency, it is possible to produce images of 3D systems which provide true volumetric visu- 
alization, rather than surface projections. We have been working on this type of system for the 
past three years (Russell and Miles, 1987), concentrating on techniques which are efficient enough 
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to be used interactively on existing computer systems. Pixar Corporation has recently been devel- 
oping a package to support volumetric visualization, including an approach called Volume 
Rendering Technique, which they developed with Phillips Medical Systems and Dr. E. Fishman 
(1987) of Johns Hopkins University. This package is perhaps the most comprehensive image- 
based system commercially available at this time. 

An approximation to volumetric imaging is also provided in PLOT3D, a graphics software 
system developed at JPL. This package includes a facility for producing nested transparent con- 
tour surfaces from a volumetric data base, which provides surprisingly good visualization of the 
data. Its primary limitations are data size (about 100,000 data points) and the number of contours 
it can support. Also, since this is a rather symbolic representation, it must be interpreted with care. 


VOLUMETRIC VS. 2 1/2D VISUALIZATION 


Normal pictorial illustration (stills), and most widely used 3D graphics techniques are limited to 
providing 2 1/2D surface images. That is to say, along any line of sight there is only one object or 
surface visible. This usually produces pictures from which a rough idea of the three-dimensional 
structure of the original scene can be deduced. In contrast. X-ray images generally do not have a 
unique interpretation as projections of some three-dimensional subject, and even X-ray stereo pairs 
are insufficient to provide an unambiguous inteipretation without a priori knowledge about the 
subject. 

This is a computational constraint which applies not only to visual observation of pictures, but 
to interpretation of volumetric projections in general. Vision, however, is capable of limited volu- 
metric perception and comprehension, if given adequate stimulus. 

In order to achieve effective volumetric perception, it is necessary to present volumetric data in 
a form that vision is accustomed to dealing with. While cross sections are often useful for detailed 
study of internal features, it is difficult or impossible to fully comprehend the 3D structure of an 
object in this manner. Instead, data must be presented as we would see a real object. Natural 
visual processing transforms this information back into a mental structural model. Volumetric 
characteristics of the data are conveyed by making the projection TRANSPARENT, as implied in 
the earlier discussion. 

The requirements for volumetric perception are basically the same as for computed axial 
tomography. A set of projection images from many different viewpoints is computationally suffi- 
cient to reconstruct the internal details of a subject. Visual reconstruction has several added con- 
straints: the images must be presented as an ordered sequence of closely spaced views, and they 
must be shown at a rate of at least 8 to 10 frames/sec. These constraints are dictated by the tempo- 
ral character of visual perception. 

For perception of volumetric structure (rather than surface structure), complex optical phenom- 
ena such as lighting and shading, specular (surface) reflections, and diffraction and diffusion are 
not useful. In fact, these effects generally make the basic structure of volumetric scenes more dif- 
ficult to understand, overwhelming the viewer with fine details and optical distortions. Simple 
luminance and opacity are adequate for volumetric visualization. 
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SYSTEM IMPLEMENTATION 


We have developed a system at Princeton which implements this approach to volumetric visu- 
alization on a PC/AT (Russell and Miles, 1987). The algorithms upon which it is based are effi- 
cient enough to provide a usable off line visualization system on the AT (precomputed unages take 
approximately 1 min/view for 2 million data points) and they are suitable for development into a 
real-time interactive visualization system using current state-of-the-art commercial hardware 
(AT&T Pixel Machine, for example). 

The model for the system has the following characteristics. 

1 . Data consists of samples on any regular 3D lattice (e.g., simple cubic, face-centered cubic, 
hexagonal close packed). 

2 . The data elements are treated as nebulous, fuzzy regions localized around the sample coor- 
dinates. (i.e., no subvoxel definition-consistent with proper sampling technique). 

3 . Optical model includes luminance and opacity control at each data point, with the possibility 
of handling a light source (no refraction or specular reflection). 

4. Views are computed directly from the data, without any intermediate representation. This 

reduces the risk of artifacts and avoids simplification of the data that may lead to the loss of 
features. 

5. Perspective is not supported (this is subordinate to motion). 

This combination of characteristics yields a model which is well-behaved and computationally 
efficient, with enough flexibility to provide a broad range of visual effects. 

The implementation on the PC/AT operates in a two-step process. For a given data base, a 
sequence of views is computed, based on a selected set of optical characteristics onto which the 
data are mapped, and a viewpoint and axis of rotation for the data. Each image takes about 60 to 
75 sec, for a typical data base of 2 million samples (e.g., 32x256x256 or 128x128x128), and we 
usually generate anywhere from 15 images (for a restricted range of views) to 120 images (for a 
full rotation of the data). The images are stored on a disk as they are generated. When a sequence 
is complete, the images are loaded by a second program for viewing. Up to 180 clipped images 
(176x176) may be loaded into 6 Mbytes of RAM on the PC/AT. They may then be viewed as a 
movie on a full-color, 8-bit greyscale display at frame rates up to 15 frames/sec. The viewpoint is 
controlled interactively using a mouse, within the precomputed range. 


EVALUATION 


This method of visualization provides good comprehension for a range of subjects and optical 
characteristics. Its most significant advantage is that it is very robust. There is little or no prepro- 
cessing of the data, so there are generally no computational artifacts. Even data containing no 
distinct surfaces can be accurately visualized, since this method does not rely on surfaces as the 
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fundamental elements of a scene. The use of motion as the means of communicating structure 
allows all the data to be made visible through the use of transparency. This provides a high degree 
of confidence in the resulting visualization. It is also robust in the sense that an informative set of 
images can be produced using simple optical characteristics (luminance = data value, high trans- 
parency) with little or no a priori knowledge about the data itself. 

The motion/transparency approach is most effective with scenes of moderate complexity (such 
as that shown in Fig. 1), that is, scenes whose structure can be largely comprehended as a whole. 
With very complex scenes, containing perhaps hundreds of detailed components (e.g., a video 
cassette recorder guts), this type of visualization suffers from showing too much information, 
which cannot be fully comprehended as a single entity. 


COMPLEXITY 


The issue of complexity arises in visualization for two distinct reasons. The first is the visual 
limitation just mentioned. The mind is incapable of performing a complete internal reconstruction 
of a volumetric scene, as is done in a CAT scan, for example. We have observed that beyond a 
certain level of complexity in depth (apparently three to four layers of structure), the mind's ability 
to maintain a conceptual model of a scene begins to fail. 

In addition to the visual/conceptual limitation, there is an optical constraint which limits the 
degree of complexity which is practically acceptable. There is a tradeoff between the amount of 
transparency used (which affects the visibility of embedded structures) and the amount of contrast 
available in small features. This is directly related to signal-to-noise (S/N) ratio. Vision does not 
have particularly large S/N ratio, so fine details quickly lose definition as transparency is increased. 
This is also a limiting factor in CAT scans, but the devices used have much higher S/N ratios, so 
much lower contrast can be tolerated in CAT-scan source images than is detectable visually. 

These considerations provide strong motivation to develop means of reducing and controlling 
the level of complexity in volumetric visualization. 


THE ROLE OF BINOCULAR VISION 


From a very early point in our investigation of visualization, it was clear that stereo pairs were 
inadequate as illustration of volumetric scenes. Once we had a working visualization system based 
on motion, it was easy to see how much more comprehensive this approach is than static stereo 
viewing. For some time, we assumed that adding stereopsis to the motion-based system would 
not be worthwhile, since static experiments suggested that stereopsis would not work well on pre- 
cisely those scenes where some improvement was needed. Specifically, scenes with extensive 
volumetric content and high complexity, such as medical data, generally have low contrast and few 
clearly defined, unique features on which stereopsis can operate. For scenes which are visualized 
with low transparency, which provides more distinct features, stereopsis is not really needed since 
these scenes are generally quite easily understood with only the motion-based visualization. 
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When we actually were able to try out stereo and motion together, the results were somewhat 
surprising. With scenes of medical data with moderate to high transparency, static stereo viewing 
is relatively ineffective, as expected. However, when motion and stereo viewing are used 
together, the stereopsis provides noticeable enhancement to the visual perception of the structure 
over motion alone. There is apparently some interaction between the visual mechanisms which use 
stereo and motion to deduce structure. The combined effectiveness suggests that stereopsis is 
facilitated by information made available by motion, which perhaps allows better feature matching 
between images, resulting in more and better disparity measurements. 

This strong interaction between stereopsis and motion perception means that stereopsis must be 
considered as an important part of any visualization system. Though motion is very powerful 
alone, considerable enhancement is possible through the use of binocular vision. 


CONCLUSIONS 


This approach to visualization, using transparency and motion in an image-based system, has 
significant advantages over systems based on solid rendering or graphical modeling. Most signifi- 
cant are the broader range of volumetric structure which can be visually represented and the 
robustness and freedom from artifact which volumetric visualization provides. A comprehensive 
visualization facility should certainly include the ability to perform both image-based and graphical 
rendering, and in the future these techniques should be increasingly integrated to allow both 
graphical and image-based components . a single visualization. 

Computers are now becoming available which will be capable of performing visualization tasks 
interactively. This will dramatically change the way in which visualization is used, particularly for 
very complex subjects. As interactive visualization becomes more practical, the current emphasis 
on development of techniques for data reduction and rendering should be supplanted by the need 
for means of controlling and interacting with the visualization process. As the potential degrees of 
freedom for controlling a visualization increase with the complexity and size of scenes, the design 
of effective control mechanisms will be a difficult endeavor. 

Some simple control mechanisms, such as clipping, spatial editing tools, and 3D cursors, are 
relatively easy to implement. However, for complex data, control mechanisms should parallel the 
way in which structures are decomposed and manipulated conceptually. This means providing the 
capability to specify the structural components of a scene and control their visual characteristics by 
referring to them as objects. Automated or computer-aided object segmentation is required to make 
this practical, but for the purpose of interactive control of visualizations, the accuracy and reliability 
of segmentations need not be as high as it must for conventional, noninteractive visualization. 

Additionally, it may be useful to be able to produce geometric distortions of data in order to 
push obstructing objects out of the way without separating them altogether from the region of 
interest. The net effect would be to produce the equivalent of an exploded view for structures of 
nondiscrete components. This would be particularly useful in medical applications. If information 
about connectivity and stiffness can be incorporated into the process, this could make the visual- 
ization system even more useful in surgical training or preoperative planning environments, where 
the mechanical properties of tissue structures is very important. 
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Advanced modes of interaction will become more and more important as volumetric display is 
applied to more ambitious problems of data interpretation. 

This work was supported by Princeton University School of Engineering. Additional support 
for G. Russell was provided by the Office of Naval Research through their Graduate Fellowship 
program. 
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Figure 1 — Vortex rings resulting from the Crow instability. Navier-Stokes simulation data pro- 
vided by Dr. Micheal Shelley, Princeton University. 
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FINAL PROGRAM 


Spatial Displays and Spatial Instruments: 

A Symposium and Workshop 
Sponsored by the 

National Aeronautics and Space Administration 

and the 

University of California, Berkeley 

August 31 — September 3, 1987 
Asilomar, California 


August 31 

2:00-5:00 pm Check-in and Orientation 

4:00-5:00 pm Reception 


6:00 pm Dinner 

7 :0O-8 :00 pm Welcomes 

D. Nagel 

Chief: Aerospace Human Factors 
Ames Research Center 


Conference Purpose 

Pictorial Communication 
S. R. Ellis 

Ames Research Center 

The Role of Pictorial Communication in Aerospace 
M. W. McGreevy 
Ames Research Center 


Conference Logistics, etc. 

M. Moultray and S. R. Ellis 


September 1 

7:30-8:20 am Breakfast 

8 :20-l 2:30 pm Invited Paper Session 1 

Chairman: S. R. Ellis 
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SPATIAL PERCEPTION 


8:30 am “Perspectives on Perspective” 

Professor R. L. Gregory 
University of Bristol Medical School 
Introduction by S. R. Ellis 

5 -min discussion 

9:05 am “Visual Realism in Boeing Simulators” 

C. Kraft 

Formerly Boeing Commercial Aircraft Company 
Introduction by J. Cutting 

5-min discussion 

9:40 am 10-min Coffee Break 

SPATIAL ORIENTATION 

9:50 am “Perception of Egocentric Visual Direction” 

Professor I. Howard 
York University 
Introduction by H. Mittelstaedt 

5-min discussion 

10:25 am “Egocentric Direction in Simulators” 

T. Furness 

Wright-Patterson Air Force Base 
Introduction by S. Fisher 

5-min discussion 

PICTURE PERCEPTION 

1 1 :00 am “Picture Perception and Virtual Space” 

Professor H. A. Sedgwick 
SUNY College of Optometry 
Introduction by J. Perrone 

5-min discussion 

1 1 :35 am “The Design of Pictorial Displays” 

Professor S. Roscoe 
New Mexico State University 
Introduction by J. Hartzell 

5-min discussion 
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12:20-1:30 pm Lunch Break 


1 : 30-5 : 50 pm Contributed Paper Session 

Chairman: M. Kaiser 

SPATIAL PERCEPTION 

1 :30 pm “Spatial Factors Influencing Stereopsis 
and Fusion” 

Professor C. Schor 
U.C. Berkeley 

1 :50 pm “Scaling Stereoscopic Space” 

Professor J. Foley 
U.C. Santa Barbara 

2:10 pm “Paradoxical Monocular Stereopsis and 
Perspective Vergence” 

Professor J. T. Enright 

Scripps Institution of Oceanography 

2:30 pm “The Perception of Three 

Dimensionality Across Continuous 
Surfaces” 

Professor K. Stevens 
University of Oregon 

2:50 pm “Perceiving Environmental Properties 

From Motion Information: Minimal 
Conditions” 

Professor D. Proffitt 
University of Virginia 
and 

M. Kaiser 

Ames Research Center 

SPATIAL ORIENTATION 

3:10 pm “Memory Distortions of Visual 

Displays” 

Professor B. Tversky 
Stanford University 

3:30 pm 20-min Coffee Break 
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PICTURE PERCEPTION 


3 :50 pm “The Effect of Changes in Viewpoint 
on the Pictorial Perceptions of Spatial 
Layout and Orientation Relative to the 
Observer” 

Professor B. Goldstein 
University of Pittsburgh 

4: 10 pm “Cinematic Efficacy, or What the Visual 
System Did Not Evolve to Do” 
Professor J. Cutting 
Cornell University 

4:30 pm “Congruence Under Motion as a Basis 
for the Perceived Geometrical Structure 
of Forms andSpaces” 

Professor J. Lappin 
Vanderbilt University 
and 

Dr. T. Wason 
ALLOTECH 

4:50 pm “A Theoretical Analysis of the 

Recognition of Pictorial Displays” 
Professor I. Biederman 
SUNY Buffalo 

5:10 pm “Spatial Displays and Spatial 

Instruments from the Graphics Design 
Perspective” 

A. Marcus 

Aaron Marcus Associates 

5:30 pm “Interactive Displays in Medical Art” 
Professor D. McConathy 
University of Illinois 

6:00-7:30 pm Dinner 
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8:00-10:00 pm 


Poster Sessions and Informal Discussion 


“Synthetic Perspective Optical Flow: Influence on 
Pilot Control Tasks” 

T. Bennett, W. Johnson, and J. Perrone 
Ames Research Center 

and 

A. Phatak 

Analytical Mechanics Associates 
Ames Research Center 

“Visual Enhancements and Control in Telerobotics” 
W. S. Kim, F. Tendrick, and Professor L. Stark 

U. C. Berkeley 

“Visual Slant Underestimation” 

J. Perrone and P. Wenderoth 

Ames Research Center and University of Sydney 

“Optical and Gravitational Information in the 
Perception of Eye Level” 

Professor A. Stoper and M. Cohen 
Ames Research Center 

“Interactive Spatial Instruments for Proximity 
Operations” (video) 

Professor A. Grunwald and S. R. Ellis 
Ames Research Center 

“Exocentric Direction Judgements Based on Pictorial 
and Real-World Layouts” (video) 

S. R. Ellis, Professor A. Grunwald, and S. Smith 
Ames Research Center 

“Criteria for the Successful Representation of 
Information” 

Professor M. Hagen 
Boston University 

“Development of a Stereo 3-D Pictorial Primary 
Flight Display” (video) 

M. Nataupsky 

Langley Research Center 

and 

T. Turner, H. Lane, and L. Crittenden 
Research Triangle Institute 


49-5 


“Representational Structure for Evaluation of 
Human/Robotic System Control” 

K. Corker 

BBN Laboratories Incorporated 

“Adaptation to Non-Zero Disarrangement of the 
Visual Field” 

Professor R. Welch and M. Cohen 
Ames Research Center 

“Theoretical Issues in the Development of a 2-D and 
3-D Computer-Aided Designer Support System” 

J. Hartzell 

Ames Research Center 

“Telepresence in Dataspace” (video) 

S. Fisher 

Ames Research Center 

“The Photo-Colorimetric Space as a Medium for the 
Representation of Spatial Data” 

K. F. Kraiss and H. Widdel 
Forschungsinstitut fur Anthropotechnik 

“The Role of Attensity in Spatial Perception” 

M. Companion 
Lockheed-Georgia Company 

“Achieving a Concrete ‘UP’: Embodiment of Spatial 
Relationships in a Head-Mounted Display System” 
(video) 

W. Robinett 
Ames Research Center 

“Requirements and Features of a Synesthetic 
Supermedium” 

Professor R. Mallar 
ATARI Computer 

Arstechnica: Center for Art and Technology 
University of Massachussetts 

“Helmet Mounted Displays — Spatial Orientation 
Problems” 

S. Hart 

Ames Research Center 
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“How to Reinforce Perception of Depth in Single 
Two-Dimensional Pictures” 

S. Nagata 

NHK Science and Technical Research Laboratory 

“Direction of Movement Effects Under Transformed 
Visual-Motor Mappings” 

H. Cunningham and Professor M. Pavel 
Stanford University 

“Efficiency of Graphical Perception” (video) 

Y. Gu, Professor G. Legge, and A. Luebker 
University of Minnesota 

“Applications of Human Factors for Cartography and 
Geography” 

Professor George F. McCleary 
University of Kansas 

“Interactive Digital Video Interface to an Atlas of 
Histology” 

Michael D. Doyle 

University of Illinois, Urbana-Champaign 


September 2 

7:30-8:30 am Breakfast 

8:30-12:00 am Invited Paper Session 2 

Chairman: S. R. Ellis 

MANIPULATIVE CONTROL 

8:35 am “Visuo-Motor Plasticity and Time Lags” 

R. Held and N. Durlach 
MIT 

Introduction by D. Fadden 
5-min discussion 

9 : io am “Displays and Controls for the Space Shuttle Arm” 

G. M. McKinnon 
CAE Electronics Ltd. 

Introduction by B. Bridgeman 

5-min discussion 

9:45 am 10-min Coffee Break 
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VESTIBULAR ASPECTS 


9:55 am “Theories of Visual- Vestibular Interaction” 

C. Oman 
MIT 

Introduction by R. Haines 
5-min discussion 

10:40 am “Vestibular Realism in Simulators” 

J. Sinacori 
Consulting engineer 
Carmel, California 
Introduction by E. Palmer 

5-min discussion 

COMPUTER GRAPHICS 

1 1 : 1 5 am “Graphics Hardware and Software: Coming 

Attractions” 

F. Baskett 

Silicon Graphics Inc. 

Introduction by (to be determined) 

5-min discussion 

1 1 : 50 am “The Making of the Mechanical Universe” 

J. F. Blinn 

JPL Graphics Laboratory 
Introduction by M. Kaiser 

5-min discussion 

12:30-1:45 pm Luncheon 

Speaker: J. P. Allen 
Space Industries Inc. 

(former Shuttle astronaut) 

“The Challenges of Flying the Manned Maneuvering Unit in Earth Orbit” 

1 :50-5:30 pm Contributed Papers 

Chairman: A. Grunwald 

MANIPULATIVE CONTROL 

1 :50 pm ‘Two Modes of Visual Representation” 

Professor B. Bridgeman 
U.C. Santa Cruz 
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2 ; i o pm “Perception- Action Relationships Reconsidered” 

Professor W. Shebilske 
Texas A&M University 

2:30 pm “A Computer Graphics System for Visualizing 

Spacecraft in Orbit” 

D. Eyles 

Charles Draper Laboratories 

2:50 pm “Displays for Telemanipulations” 

B. Hannaford, M. Salganicoff, and A. Bejczy 
Jet Propulsion Laboratory 

3 : io pm “Experience in Teleoperation of Land Vehicles” 

D. McGovern 

Sandia National Laboratories 

3:30 pm “Spatial Displays and Pilot Control: Where Do We 

Go From Here? 

D. Fadden, R. Braune, and J. Wiedemann 
Boeing Commercial Airplane Company 

3:50 pm 20 min Coffee Break 

VESTIBULAR ASPECTS 

4: 1 0 pm “Determinants of Space Perception in 

Weightlessness” 

Professor H. Mittelstaedt 

Max Planck Institut fur Verhaltensphysiologie 

4:30 pm “Voluntary Presetting of the Vestibular Ocular Reflex 

Permits Gaze Stabilization Despite Perturbation of 
Fast Head Movements” 

Professor W. Zangemeister 
Neurologische Klinik der Universitat 
Hamburg 
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COMPUTER GRAPHICS 


6:00-8:00 pm 
September 3 

7:30-8:30 am 
8:30-9:00 am 

10:00 am 
11:30 am 
11:30-12:30 pm 
12:30-2:30 pm 

2:35 pm 
4:00-4:15 pm 


4:50 pm “Wide Angle Display Developments by Computer 

Graphics” 

W. A. Fetter 
Siroco 

5:10 pm “V isualizing Space Filling Data” 

G. Russell 
Princeton University 

BBQ on Asilomar Terrace 


Breakfast 

Summary Session 

Summary: L. Stark/U.C. Berkeley 

Thanks to all 

Checkout 

Leave for tours of Ames Research Center 
Arrive Ames Research Center 

Lunch at Ames Cafeteria (not included in conference fee) 

Open House at Aerospace Human Factors Division and possibly 
Vestibular Research Facility 

Leave for Return to Asilomar 

Arrive Monterey Airport/Asilomar Conference Center 
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